This interface method, and the ones on ScalarFunction and SqlInvokedScalarFunction, are parameters to an annotation. Does a user implementing a function using the Presto SPI manually specify this value in the annotation? If not, then these should be removed.

Yes they are specified in the annotation. If it is not specified, then it is -1 by default

Annotation looks like this

@CodegenScalarFunction(value = "function_name", calledOnNullInput = true, pushdownSubfieldArgIndex = 0)

Can you add some documentation on how this needs to be used to our documentation?

Also, why not add an annotation to the argument itself? Something like, @RowMayBeDereferenced. (BTW, can you give an example of how a function could know that this is safe to do?) You could annotate multiple of them, and we we could validate that the argument is, indeed, a struct to begin with.

Also, your example reference internal queries, can you add pastes of the explain plans?

Can you also add some tests?

Instead of using annotation on the argument, I have chosen to pass the argindex in the codegen decorator because this allows it to be inside the FunctionMetadata.

I added some tests to TestHiveLogicalPlanner. One more change I will make is to perform some validation that the argIndex specified does correspond to a rowtype. And throw a warning when the code path is not reached due to invalid index

@tdcmeehan @rschlussel
Looking into the ComplexTypeDescriptor approach. I think it requires some changes to FunctionExtractor. reshape function is registered through a plugin. Currently, the Plugin interface only supports annotated function definitions.

Tried adding this line in FunctionExtractor, but this apporach does not seem ideal. It also assumes that the instance of SqlScalarFunction only has one constructor and it's an empty constructor. The way that other SqlScalarFunction get registered is inside BuiltInTypeAndFunctionNamespaceManager. And they are all BuiltInFunction because SqlScalarFunction is extension of BuiltInFunction.

        if (SqlFunction.class.isAssignableFrom(clazz)) {
            try {
                return new FunctionListBuilder()
                        .function((BuiltInFunction) clazz.getConstructor().newInstance())
                        .getFunctions();
            }
            catch (Exception e) {
                throw new RuntimeException(format("Error adding BuiltInFunction %s", e.getMessage()));
            }
        }

I also don't see any examples of SqlScalarFunction added through plugin. If you look at any Plugin implementation, the getFunctions override only adds functions that are annotated, most of them are with @ScalarFunction. Take for example I18nFunctionsPlugin.

I'm wondering if you two have any other thoughts on this approach and other things we need to consider.

I agree that ComplexTypeFunctionDescriptor would be the best place for it. However, SqlScalarFunction is in presto-main-base. You can't use it from a plugin. You need a way to generate it from an annotation.

What if we added the ability to specify the complexTypeFunctionDescriptor from an annotation. like what was removed in this commit d6e008c (and added earlier in the same PR). I can't remember why it was removed.

I see the problem now. Plugin-registered functions must use annotations and this function is registered through the Plugin SPI, which means we can't incorporate the feedback above directly (for example, from within a Plugin you won't even be able to subclass SqlScalarFunction).

So taking a step back, the main thing I'm not in favor of in this PR is adding pushdownSubfieldArgIndex to our scalar function annotations (going back to scope of API addition vs. what we get out of it). We also already use ComplexTypeFunctionDescriptor to instruct PushdownSubfields on how to interact with complex types, but these all apply to maps and arrays, and they're functions that are built-in and can use the verbose SqlScalarFunction definition method above because they're defined in the main module.

I can think of a few of ways of going about it.

1. Trino has a FunctionProvider SPI that lets you lazily build scalar functions, that could work to allow us to verbosely construct this internal function from within the SPI. (Similarly, we could allow SqlScalarFunction to annotate a no-arg static factory method which returns a concrete instance of SqlFunction).
2. We could add an annotation that allows one to define companion information in the function, such as ComplexTypeFunctionDescriptor. Or we could add a separate annotation that allows the fields in ComplexTypeFunctionDescriptor to be specified independently, though this would be awkward at the method level, and I think it would be more appropriate at the argument level.

I'm also wondering, looking at ComplexTypeFunctionDescriptor, if the functionality we're adding here can already be accomplished by the code in this class, specifically the outputToInputTransformationFunction field.

I would be in favor of restoring an annotation, but I think the one above was mostly a 1:1 copy of ComplexTypeFunctionDescriptor and can be improved (I don't think it should have a StaticMethodPointer that references internal engine classes, and argumentIndicesContainingMapOrArray should just be reworked to be per-argument).

okay, so i think the next steps here are
1.Add the new passthrough field to ComplextTypeFunctionDescriptor instead of scalar annotation. can then use it in SubfieldPushdown
2. create an annotation for creating a complexTypeFunctionDesscriptor (probably doesn't need to be fully featured at this point? can add to it as needed), add parsing ability for the annotation for generating the function metadata.
2. use the new annotation in fb_reshape_row

or has no typeConstraintMapping entry (maybe no constraint on the type) or typeConstraintMapping value is null and TypeVariableConstraint.nonDecimalNumericRequired is false (i.e. there's no constraint preventing row type?) ?

I don't think this is too restrictive. The argument index is only checked for these conditions if the pushdownSubfieldArgIndex.isPresent().

Also, the typeVariableName should always be present because signature.getArgumentTypes() always contains the same number of parameters as the function signature. There is always a type constraint imposed on a positional argument.

I think you might have it confused with typeVariableConstraints. typeVariableConstraints might be empty if we don't use an alias for a type (T for RowType). typeConstraintMapping is built using typeVariableConstraints.

So my check ensures that the parameter is either directly named a "row" type, or it has a type alias that maps to a "row" type

In summary: typeVariableConstraints is retreived from this annotation: @TypeParameter(value = "T", boundedBy = ROW)

argumentTypes is retrieved from the function definition public static Block customStructWithPassthrough(@SqlType("T") Block struct)

To make it clearer, heres an example of what the variables look like when the function takes in more than one parameter

Is it possible for something of a generic type "T" to accept a row type, but not have variadicBound "row". E.g. maybe it can be used for all types , so it doesn't have a TypeVariableConstraint, or it just requires that the types be orderable, but can be a row or other type (so variadic bound is null)?

Is this the logic desired?

            // The type variable must be directly a ROW type
            // or (it is a type alias that is not bounded by a type)
            // or (it is a type alias that maps to a row type)
            boolean meetsTypeConstraint =  (!typeConstraintMapping.containsKey(typeVariableName)  && typeVariableName.equals(com.facebook.presto.common.type.StandardTypes.ROW)) ||
                    (typeConstraintMapping.containsKey(typeVariableName) && typeConstraintMapping.get(typeVariableName).getVariadicBound() == null && !typeConstraintMapping.get(typeVariableName).isNonDecimalNumericRequired()) ||
                    (typeConstraintMapping.containsKey(typeVariableName) && typeConstraintMapping.get(typeVariableName).getVariadicBound().equals(com.facebook.presto.common.type.StandardTypes.ROW));

yeah, that looks good

same comment about maybe this is too restrictive and need to add some other conditions.